Heater profile overview
Info

This file contains the overall goals and context for the project, see Heater profile technical documentation for a more in depth guide to get the code running

Solar Orbiter magnetometer

Solar_Orbiter_instruments_pillars-558077216.png

The Solar Orbiter magnetometer has two instruments, IBS and OBS, which sit on the boom. The aim is the measure the interplanetary magnetic field, using a fluxgate magnetometer, which is on the order of nT.

This instrument is extremely sensitive, and can measure very small fluctuations in the solar wind. Unfortunately, it can also pick up the magnetic field of the other instruments aboard the spacecraft. Since space is very cold, each instrument has a heater to keep the electronics within a certain temperature window. If the temperature drops below a certain threshold, the heater will turn on. There is a separate code that removes the heater signals from the hundreds of heaters on the spacecraft.

Heater interference

The largest interference signal comes from the magnetometers' own heater, which turns on roughly every 15 minutes for 1 minute at a time. The signal looks like this: Pasted image 20230807115906.png

Obviously, an interference on the scale of 1nT is not great when the real solar wind can be on the order of 10nT. You can clearly see how the heater turns on really sharply at 0 seconds, stays on, then turns off sharply at 1 minute.

Crucially, the heater continues to interfere with the magnetic field well after it has turned off.
Info

Radial-Tangential-Normal (RTN) coordinates are defined such that the +X axis points from sun center through the satellite (radial), and the +Z axis points along the projection of the solar North pole (normal). The +Y axis completes the right-handed coordinate system, and points towards the solar West limb (tangential) (source)

Even more frustrating is that fact that the heater profiles change over time, with no obvious cause.

Pasted image 20230807124342.png

The redder the colour the later the date of the profile

Could this physics be included in a model?

Average profiles

To remove this signal, the current solution takes an average profile from all the 15 minute intervals within a 3 day period. This works reasonably well, however, there are some major issues:

  1. This method doesn't work when the solar wind magnetic field is large, hard to isolate the 1nT signal
  2. Relies on the assumption that the solar wind fluctuations average to zero, which is not always true
  3. We don't know why the heater profile changes
  4. Heater profile can drastically change at midnight when a new set of days are used

All profiles from 2022, can see that the ones near perihelion (red) are just nonsense Pasted image 20230807124420.png

Suggestion

Instead of stepping the window for the average by a day, move it by 15 minutes. This will allow a much smoother change in heater profile, rather than a step change at midnight

Goals

Therefore, the goals of this project are to:

  • Create a model that can predict the shape of the profile
  • Apply this to times when the averaging does not work
  • Understand why the profile changes, better understand the thermal environment around the instrument

To help with these goals, there is a wealth of data about the state of the spacecraft, known as house keeping data. This keeps track of everything that happens on the spacecraft, such as:

  • currents drawn by instruments
  • temperature of instruments
  • angle of solar arrays
  • distance from the Sun

and many more...

Comparing profiles by date

To better see the systematic changes in heater profiles, we often plot the profiles as a 2D histogram.Pasted image 20230807124823.png

The left panel shows the heater profiles, where a redder colour indicates a later profile date. We have taken these profiles and stacked them vertically, equating the value of the component with the colour scale.

To get your eye in, notice that the blue profiles on the left end at 5 minutes (300 seconds), which equals the shorter part of the histogram.

Plotting in this way allows systematic effects to be seen more easily.

For example, looking at the Bt component (second panel on the right figure). We can see that when the profile length changes from 300 seconds to 800 seconds, there is a decaying signal at around 200 seconds profile time.

Warning

This signal persists for 2 weeks, and is thought to be due to residual heat flux after the instrument changes operational temperature range.
The heaters are turned on if the temperature drops below a certain threshold. The MAG team usually runs this at around , but have used for certain periods. When the range drops from to , this effect appears.

Framing the problem

This problem sounds like a good use case for supervised machine learning, where the model is trying to learn a known answer.

Question

But how should we set this problem up?

So far, we have tried to learn how to predict the average profiles that are created for each day of data. We question whether there is a better method in this section.

Predict whole profile

Solar Orbiter was launched in February 2020, therefore we have on the order of 900 heater profiles to learn from.

However, each heater profile runs for around 15 minutes. If we tried to predict just one magnetic field component at 1 second intervals, we need to predict values each day.

Therefore, there is probably not enough data (900 days) to train a model to directly predict a vector of size 900 (15 minutes in 1 second intervals).

Suggestion

This final statement might not be true if we changed the averaging method to create a profile for each 15 minute interval instead of each day. Now we would have profiles to learn from

Regression problem

Inspired by this kaggle competition, we have turned this problem into a regression problem. By stacking all the profiles together, we are now trying to just predict one y value at a time.

Important

In other words, we are using Time through the profile as a feature to train the model.

This simple perspective shift means we now have points per profile and days, giving data points to train on, with only 1 output to predict for each component.

As you will see later, we originally tried to train a separate model for each magnetic field component. However, we then decided that it would make more physical sense to have a single model that predicted 3 outputs.

Predict each individual profile

Warning

The problem with previous two methods is that the best they can do is match the average profiles. Therefore, the model will inherit all the same problems from the averaging technique

Instead of learning what the average profiles look like, we could try and predict the shape for all the 15 minutes individually.

Machine learning needs to minimise some value, like the mean square error. Therefore, we just need to devise some function to be minimised... This is easier said than done though, and has not been explored yet.

Outlining the model

The rest of this document outlines the steps taken and current outputs of the model, as well as where we could go next.

Again, the technical implementation details are outlined in Heater profile technical documentation

Overfitting

Warning

The main challenge with this problem is to train the model to learn from, rather than memorise, the training data

The model is exceptionally good on the trained data, but can hallucinate when used on unseen data. This is because the model can try and memorise the profiles, instead of learning the relationships.

This challenge is the primary motivator behind many of the techniques outlined below.

House keeping data

The current house keeping data used for the model. We have tried to include why we think these features would affect the heater profiles.

The house keeping data are grouped into different files, which is reflected in the subheadings here.

Since we have framed the problem to predict a heater profile per day, we need to also calculate the house keeping data per day, i.e. even though temperature can change many times per day, we can only give the model 1 value per day.

Suggestion

Need to add more house keeping data, however, this creates more features which seemed to confuse the training. We have tried to address this with feature selection

Distance from the Sun

Other instruments start to turn on at given times, there is more activity at perihelion. Maybe temperature of instrument depends on distance from the Sun?

Called Radius in the features, since it comes from describing the position of the spacecraft in longitude, latitude and radius. To see how this is calculated, look in Heater profile technical documentation

Pasted image 20230808171819.png

th_mag

Question

Since the heater interference is probably driven by thermal gradients, useful to know the temperature of the instruments

Temperature of the IBS and OBS instruments. Top panel shows the raw data, the second panel is a 1 day average (since we need 1 house keeping data point per day)

3rd panel is the difference between OBS and IBS temperatures

Pasted image 20230808172228.png

The rest of the features are trying to capture the decaying Bt signal discussed earlier. You can see that the OBS temperatures are either or , we want to capture the time since the temperature dropped (time_since_T_high) and also how long the temperature was run at (time_it_was_high).

This worked pretty well, but we also added another feature (cumulative) to capture how the thermal gradient would reduce over time. Ideally this would decay exponentially, but is a lot easier to just say it is linear for now.

Solar array (SA)

Question

Angle of the solar arrays could reflect sunlight onto the heater

First panel shows the raw data (blue) and the 1 day average (orange)

Second panel tries to capture the change in angle:

  • 0 no change
  • 1 increase
  • -1 decrease
  • 2 both an increase and decrease within a single day

Pasted image 20230808173104.png

High Gain Antenna

Question

The changes in these components are the same as the solar array encoding above

Pasted image 20230808173403.png

th_rpw

Question

RPW is another instrument on Solar Orbiter that sits in between IBS and OBS.

We have the temperature of its search coil magnetometer (SCM), and the change in this value.

Pasted image 20230808173646.png

MAG PWR

I assume this stands for MAG power and Amp (current), needs checking

The Mag A component had many drop outs (where it went to 0). We have tried to capture this with the No_times_A_off parameter which is the log(number of times off per day)

Pasted image 20230808173703.png

Duty cycle

Question

Shows when the heater turns one, which is normally every 15 minutes

You can see that the heater changed (blue to pink) on 2021-11-08.

Pasted image 20230811145502.png

Pasted image 20230811153317.png

Future things to include

  • Temperature of the boom
  • Mag electronics box temperatures (hk-50020)
  • Search coil heater current
  • EAS temperatures
  • EAS on or not
  • NIM01026_ENG 8V Current (16 bits)

Bad profiles

The profiles near perihelion are not usable and would hinder the model's training. Therefore, we have removed the times listed in bad_dates.csv (identified by eye)

start,end
2020-10-26,2020-11-06
2021-01-15,2021-03-12
2021-05-29,2021-06-03
2022-02-17,2022-04-20
2022-09-12,2022-11-07
2022-11-07,2022-11-08
2023-03-25,2023-05-05
If the model allows it, you could weight each day's profile by how much you believe it. For example, if the solar wind variability is high, the averaging method will not work. Therefore, the weights could be equal to the variance of solar wind fluctuations in that day

Feature manipulation

Cardinality

You may have noticed that the features above look a bit step-like. This is because we have only allowed the features to take a certain number of values, e.g. 0.1 and 0.14 would both be taken as 0.1. The technical term for this is reducing the cardinality.

This helps step overfitting, since the model cannot work out which combination of values corresponds to which day, i.e. only one day might have the value 0.012312041243 while another day had 0.0123014035407564, so the model just memorises the unique values.

This is done by helpers.reduce_cardinality(features, feature_name, N, bins)

History

Perhaps the change in profile is because something happened in the previous day. To capture this, we have shifted the data and saved in a new column Feature-name 1 where 1 means 1 day in the past.

Obviously, this increase the number of features, which means the model can have a harder time finding the real relationships. So only use this on features you think it would be relevant for
Experiment with the number of days back, we have gone for 4

Scaling

When researchers are coming up with these machine learning algorithms, they often use some basic assumptions, such as the data is a Gaussian centred on 0. Therefore, we have to scale our data to match their predictions. This is easily done in sklearn.

Custom time steps

To save computing time, I downsample the heater profiles when learning. When making the models we have two parameters to control the time steps:

  • ds_duration for the length of time through the profile (ds is downsample)
  • ds_period for the difference between time steps

Towards the end of the project, we switched to using custom time steps. This allows us to have a finer resolution in the first minute or so, then only every 5 seconds after that. The exact function for this is helpers.get_var_time_bins(ds_duration)

In the code, we call this by setting ds_period = -1

Heater 1 or 2

As mentioned in the calculate of duty cycle, the instrument switched which heater it used on 2021-11-08. We found that the profiles look distinctly different after this switch, so the best results only train on the data after this date.

A model should be able to learn the difference in signatures, which could be something to try in the future. It is harder to get good training results though

Feature selection

As we added more features, we noticed the models struggled to train as there was too much noise in the features. Therefore, we now use Recursive Feature Elimination (RFE) which recursively trains a model and removes the least relevant features.

Gradient Boosted decision tree

Most of the research was done with LightGBM, since it is lightweight and fast. There is an API you can use to fit well with sklearn.

This is a type of gradient boosted decision tree. In essence, this uses lots of weak leaners (small decision trees) to build a more complex model. In this method, the next tree is generated to minimise the error from all the previous trees.

In contrast, random forest is a collection of randomly generated trees that all vote on the answer collectively.

Early stopping

The main appeal of gradient boosting is that we can control when it stops learning, using a technique called early stopping. If we give the model a completely separate validation set, it will stop learning when the model doesn't improve the score on the validation data within N new trees.

However, this model can only learn to predict one component at a time, which is not ideal when trying to predict a vector in our problem.

Random Forest

Instead, we then opted to use a random forest, which can learn to predict a vector output. Unfortunately, this does not have the early stopping capability, and instead you have to try and stop overfitting by tweaking the model hyperparameters.

I couldn't really get this to work, which makes the random forest look worse than LightGBM in the current state of the project. However, this is what I would focus on

Conclusion

This document has hopefully outlined the general ideas behind the project. See Heater profile technical documentation for a more in depth guide to get the code running
.
.
extra space to fix a glitch in the html...